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Abstract 

A comparative study was conducted to compare two 
approaches to engineering design curriculum across different 
schools (inter-school) and across two curricula Project Lead 
the Way and Engineering Projects in Community Service 
(inter-curricula). The researchers collected curricula material 
including handouts, lesson plans, guides, presentation fdes, 
design descriptions, problem statements, and support guides. 
The researchers conducted observations in the classrooms to 
collect qualitative indicators of engineering/technology 
reasoning, collect data on the nature of students’ questions, 
how students define problems, and operate within the 
constraints of a design problem. 
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Observational protocol studies were conducted on 
students participating in Project Lead the Way curriculum 
programs and with students participating in Engineering 
Projects in Community Service (EPICS). Students were asked 
to work through an ill-defined problem, in this case the 
problem of creating a new playground for an elementary 
school. The data from these protocols were analyzed using a 
coding process; a list of universal technical mental processes 
(Halfin, 1973) and a computer program OPTEMP, (Hill, 1997) 
to accurately record frequency and time of each mental process 
employed by the students. The data from the protocol results 
were used to identify common cognitive strategies employed 
by the students to detennine where these students placed 
greatest emphasis throughout the observational protocol. 

General findings indicated that participants in the 
EPICS-High program were more solution-driven problem 
solvers, where the Project Lead the Way participants were 
generally problem-driven as defined by Kruger & Cross 
(2006). Additionally, the participants in both groups had 
completed advanced courses in mathematics, very little 
mathematics was employed (less than 3%) to describe 
constraints of the problem or predict results of proposed 
solutions. Over half of the students became fixated at some 
point on the provided picture. (Smith; Ward; & Schumacher, 
1993). This study provides important insight about how 
students solve ill-defined problems, providing vital information 
for technology education as it seeks to implement engineering 
design. 


Introduction 


The purpose of this study was to better understand the 
current status of engineering-focused curriculum programs at 
the high school level and its greater impact on student learning. 
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Specifically, this study examined the implementation of Project 
Lead The Way (PLTW) and EPICS High, engineering-focused 
curriculum and its impact on students’ abilities to employ the 
engineering design processes using mathematical and scientific 
reasoning to solve engineering design problems; a construct of 
interest to K-12 engineering education and the greater STEM 
community. 

The rise in engagement of secondary engineering 
education (Douglas, Iversen, & Kalyandurg, 2004) and the 
increase in development of engineering-focused curriculum for 
grades 9-12 (Dearing & Daugherty, 2004) provide strong 
rationale for examining pre-existing secondary engineering- 
focused programs to evaluate the impact on student learning. 

Researchers in technology education have investigated 
the effects of technology education instruction on performance 
in mathematics and science for over a decade (Childress, 1996; 
Merrill, 2001). Findings indicated that engineering drawing 
and design courses have had a significant impact on student 
achievement on mathematics standardized end-of-year tests 
(Dyer, Reed, & Berry, 2006). Another study discovered that 
engineering design instruction at the middle school level 
reduced the performance gaps among certain ethnic groups and 
increased student’s conceptual knowledge of science, while 
simultaneously generating higher order thinking skills such as 
analysis and synthesis (Cantrell, Pekcan, Itani, & Velasquez - 
Bryant, 2006). These studies indicate that there is a growing 
interest in investigating how engineering-focused instruction 
impacts student’s STEM learning. 

In addition, several initiatives have started in the last 
decade to introduce engineering principles and engineering 
reasoning into grade 9-12 classrooms (Brophy, Klein, 
Portsmore, & Rogers, 2008). While PLTW and other 
curriculum programs like EPICS-High focus on different 
aspects, from engineering problem solving to service learning; 
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each of these new curriculums share the lack of a cohesive 
body of systematic and rigorous research studies investigating 
the effectiveness of their programmatic efforts. Specifically, 
there is a lack of investigation on higher order thinking, open- 
ended problem solving and the development of design and 
engineering knowledge/skills (Chaker, 2008). 

Dyer, Reed, and Berry (2006) recommended more 
research on the instruction of engineering design to accurately 
detennine what elements of instruction are most beneficial to 
student’s learning of math and science. Similarly, Sheppard, 
Pellegrino, & Olds (2008) and Brophy et al. (2008) call for 
more research on engineering efforts in high schools. This 
becomes especially pressing, since individual states develop 
specific engineering standards (i.e. Massachusetts) and specific 
teacher licensures will likely follow. As these programs 
continue to grow, there is a need to build a strong base of 
rigorous research to provide educated and specific feedback on 
how to improve existing curricula and build a cohesive 
research agenda on engineering reasoning development in the 
K-12 grade spectrum. 


Participants 

Project Lead The Way (PLTW) is a national pre¬ 
engineering program that has been implemented into a number 
of high school and middle schools. The Project Lead The Way 
program launched in 1997 based on previous work in the 1980s 
by Richard Blais at Shenendehowa Central School (Blais & 
Adelson, 1998). Since 1997, Project Lead The Way has grown 
with over 1300 schools and 175,000 students (Bottoms & 
Anthony, 2005). Currently, there are over 280 high schools in 
the state of Indiana alone implementing curricula modules of 
Project Lead The Way. Two PLTW schools were selected in 
northwestern Indiana, these schools were selected because they 
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had successfully implemented PLTW at the high school level 
and had similar student demographics. 

Engineering Projects in Community Service (EPICS) is 
a national engineering-centered, academic service-learning 
program initiated at Purdue University (Coyle, Jamieson, & 
Oakes, 2005). The main tenets of service learning are 1) 
curricular connections; 2) reflection; 3) community 
partnerships; 4) authentic, complex and ill-structured problems; 
5) addressing real needs; and 6) performance-based assessment 
(Honnet & Poulsen, 1989). Multidisciplinary team partnerships 
are fonned with local not-for-profit organizations to define, 
design, build, test, deploy, and support engineering-centered 
projects that significantly improve the organization’s ability to 
serve the community. EPICS-High (for grades 9-12) began in 
2007 bringing the engineering design concept into the high 
school environment. The program has quickly grown to 32 
active high schools across the nation with over 650 students 
served, 50% female students, 48% minority students, and over 
50% in free and reduced lunch programs. Two EPIC High 
programs were selected in northwestern Indiana and were 
chosen by their record of success in implementing the EPICS 
High program. Every effort was made to select these 
participating schools that align with the student body 
demographics. 
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Table 1. Participating School General Student Demographics 



PLTW 1 

PLTW 2 

EPICS 1 

EPICS 2 

Enrollment 

883 

1606 

1833 

883 

Graduation 

Rate 

75.7% 

92.2% 

83.8% 

75.7% 

White 

72% 

88% 

86% 

72% 

Hispanic 

25% 

5% 

7% 

25% 

Multicultural 

2% 

3% 

2% 

2% 

Asian 

1% 

2% 

1% 

1% 

Black 

1% 

2% 

3% 

1% 

Native 

American 

1% 

0% 

1% 

1% 

Surrounding 

Area 

Rural 

Urban 

Fringe 

Urban 

Fringe 

Rural 


Research Questions 

This research was guided by the following research 
questions. Since the nature of our research is comparative, the 
result to each of the questions will be compared across 
different schools (inter-school) and across curricula (inter¬ 
curricula): 

1. What are the most common elements within student 
dialogues as they define engineering, engage in student 
collaboration and class discussions when seeking to 
solve engineering design problems? Which attributes 
or elements of engineering are missing or strongly 
represented? 

2. What are common cognitive and meta-cognitive 
strategies employed by high school students as they 
work to solve an engineering design focused problem? 
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3. What elements in the engineering design problems (or 
in the curriculum unit) encourage teachers to engage 
mathematical and science curricula elements and 
support mathematical reasoning or scientific reasoning 
of their students? 


Methodology 

The research team collected curricula material from all 
participating classrooms to analyze. This included: textbooks, 
handouts, lesson plans, PowerPoint files, design descriptions, 
problem statements, and support guides. The researchers 
conducted observations in the classrooms to collect qualitative 
indicators of engineering/technology reasoning, collect data on 
the nature of students’ questions, how students define 
problems, and how they operate within the constraints of an 
engineering design problem. 

Next, a protocol analysis session was conducted with a 
group of three student volunteers from each site. Each 
volunteer was given the same design problem (the transfer 
problem), see Appendix A. Each student was asked how they 
would proceed from the given problem statement in order to 
improve the current condition described in the statement. The 
students were asked to define the problem, list all constraints 
that they impose on this problem, and describe how he or she 
would proceed to solve the problem. The participants were 
asked to verbalize their thoughts as they worked through the 
ill-defined problem (verbal protocol methodology). The 
researchers prompted participants to keep talking through the 
problem. The testing sessions were limited to thirty minutes, 
most lasted around ten to fifteen minutes. Certainly, in the 
time constraint of a testing session, a student would be unlikely 
to reach the final stages of the design process. However, one 
of the most important stages of the engineering design process 
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occurs at the onset of a technical problem: ‘framing the 
problem’ is this important stage. Experts in the field of design 
identify that framing the problem is a critical step to the design 
process which occurs as soon as the designer is presented with 
a technical problem (Dym, Agogino, Eris, Frey, & Leifer, 
2005; SchOn, 1983). The transfer problem was developed to 
share characteristics of an ill-defined, complex, and dynamic 
design problem (Jonassen, 2000). 

Analysis 

The project involves two separate analyses of artifacts, 
(1) curricula materials, and (2) results from selected students 
transfer problem analysis. Both forms of data were analyzed 
with an analytical induction framework (Bogdan & Biklen, 
1992) that similarly to the constant comparative method of 
grounded theory (Strauss & Corbin, 1990), combines data 
analysis, further literature review, and theory-building in a 
cyclical manner. In the analytical induction approach, data 
built the basis for further descriptions and interpretations, and 
in contrast to grounded theory is informed by prior research. 

As students progressed through the transfer problem 
session, the students’ cognitive processes were identified and 
coded from a list of 17 universal mental processes (Halfin 
(1973). A computer analysis tool called the Observation 
Procedure for Technology Education Mental Processes 
(OPTEMP) (Hill, 1997) was utilized to capture, record, and 
organize the codes from each transfer problem session. The 
researchers coded the actions and cognitive processes used by 
each student participant as he or she worked through the 
engineering design problem. 
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Findings 


Classroom Observations: 

A total of three class observations were conducted at 
each PLTW and EPICS-High site. Each PLTW and EPICS 
High class session was 90 minutes for a total of 270 minutes of 
observation at each site. 

Project Lead the Way Observations 

The course observed was PLTW Principles of 
Engineering. The PLTW students were working on marble 
sorting activity during the observation sessions, the activity 
required student teams to design and build a device to sort 
marbles by color using Fishertechnik parts. The nature of 
much of the classroom dialogs were regarding Fishertechnik 
parts (motors, sensors, structures, etc) and how the students 
were planning to connect the various parts to make the device 
function properly. However, as the student teams’ device 
began to be tested, malfunctions often occurred. As a result of 
these device malfunctions; students worked as cohesive teams 
to troubleshoot various problems encountered. Student dialogs 
were healthy problem solving conversations and represented 
cooperative team efforts. However, students did not appear to 
be using a systematic approach to solve problems. The 
students approach to solving problems was reactive as they 
employed trial and error approaches, which might also be 
described as “tinkering”. Students often were observed 
reconfiguring the device parts such as wires, motors, and 
sensors. 

EPICS High Observations 

Both EPICS High participating schools were observed 
for a total of three class observations. One important 
observation about both EPICS High classes was the use of an 
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engineer’s notebook. Both EPICS High programs required 
students to obtain and keep an engineer’s notebook for the 
course. The EPICS High School #1 class was discussing 
“great teams” during one of the classroom observations and 
“community” during another class observation. The 
community discussion was an EPICS High created lesson titled 
Your community...My community. The instructor led a class 
discussion by asking questions like “How big is your 
community? Where does it begin and end?” The instructor 
encouraged students to think about how they individually 
define community, what did community mean to them. Later 
in the class, the discussion led to the brainstorming of 
community service projects to be explored by the class. One 
researcher had a conversation with the EPICS #1 instructor 
who shared that his school started the EPICS High program to 
purposefully target the middle level academically perfonning 
students instead of targeting the high perfonning students. The 
course gave students an alternative to PLTW which the school 
also offers. The Purdue EPICS program provided a $48,000 
grant to provide the EPICS school #1 with new laptops for all 
the students to use in the class. The grant also allowed the 
school to travel to Aurora, IL to the Walter Payton Brew House 
roundhouse. Last year’s EPICS High project was a feasibility 
study to restore the train roundhouse that is located in the 
community. EPICS High school #1 class is 12 weeks long. 

One classroom observation at EPICS High #2 school 
found small groups of students (two groups of two students) 
discussing with the instructor their EPICS High community 
projects. The instructor reminded the students that they needed 
to be recording their design thinking in the design notebooks 
and how the information was to be recorded. The instructor 
asked the students to present their design project ideas 
informally to the group. One group selected a community 
project to design and build a walking path for students on 
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school grounds, the path would need to be wheelchair 
accessible. The design team was talking about making a 
pathway using some sort of gravel. The design discussion lead 
to brainstonning the different material they could use so the 
wheel chairs won’t get caught. The other design team selected 
a project to create a storage container that would allow teachers 
to use a magnetic key fob to lock and unlock their laptop 
computers in a storage unit. Many teachers at this school have 
complained about having to carry their laptops everywhere 
they go for security reasons instead of being able to just leave it 
in their classrooms or one central location. The EPICS High 
#2 class was small and as a result was a very informal 
classroom structure, however there was good classroom 
dialogue regarding the design process. The instructor 
emphasized the importance of creating a clear and concise 
problem statement and recording design work in the engineer’s 
notebook. The EPICS High #2 instructor was new to the 
EPICS High program and had just completed EPICS training in 
the summer of 2009. The EPICS High #2 school program runs 
the EPICS High course as a club and not an official class; 
students do not get course credit and meet after school hours. 

Curriculum Documents 

A review of PLTW curriculum documents for the 
course Principles of Engineering revealed a focus on teaching 
students about systems, subsystems, open and closed loop 
systems, basic computer programming, and troubleshooting 
problems within a system. Later in the semester, students also 
learned about basic statics and dynamics. Furthennore, 
students were taught how to use free body diagrams and 
conduct basic vector analysis. This knowledge would have 
been helpful for students who participated in the transfer 
problem. However, this instruction came after the transfer 
problem testing session. Upon review of the Principles of 
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Engineering curriculum, the researchers identify that the 
students are exposed to a limited scope of the engineering 
design process. PLTW curriculum program consists of a series 
of courses in pre-engineering; therefore it is unfair to fully 
assess students design capabilities upon their completion of just 
one course. Assessment of students who have experienced all 
the PLTW courses would be ideal such as in the PLTW 
capstone course: Engineering Design and Development. This 
course was studied in a similar protocol analysis study (Kelley, 
2008). However, the schools in this study did not offer this 
course. Furthennore, most of the students used in this study 
had taken Introduction to Engineering Design that provided a 
broader overview of the engineering design process so students 
should have been prepared to employ the engineering design 
process to solve the transfer problem. 

The EPIC High curriculum documents that were 
studied were lesson plan documents taken from EPICS High 
Curriculum Module 1- Design, week 3. The activities in this 
module focus on design notebooks. One activity required 
students to assess their current note-taking skills by examining 
their notebooks they keep for other classes. The activity asks 
students to brainstonn in a small group how the design 
notebook differs from traditional class notebooks. 

Other similar class activities included making use of 
other people’s design notes, building note taking skills, and 
building organizing skills for design notebooks. Lessons for 
later in the course focused on the topic of solving engineering 
problems in teams. These lessons required students get into 
small or large design teams and work to solve some ill-defined 
problems. One problem was as follows: 

“In your group, solve the following design problem: 

• Landfill space is rapidly running out. 
Develop a plan to eliminate your city’s 
dependence on the local landfill. 
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Imagine, for now, that the city’s 
population is 100,000. 

• As you solve this problem, each member 
should take notes according to your 
design notebook guidelines.” 

(EPICS High curriculum Module 1- Design lesson plans, p.29). 

When students were finished exploring possible 
solutions, they were required to compile a design record 
providing a design rationale including experimental 
measurements, sketches, safety precautions, design criterion 
rationale. Students were also asked to consider assessing how 
they functioned as a design team during the assignment. This 
exercise in open-ending problem solving would provide ideal 
experiences for students and should prepare students for the 
transfer problem, see Appendix A. 
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Ill-defined Transfer Problem Session: 

Table 2. Transfer Problem Participants’ Coursework 


Participant Grade Math Science Technology 

level Courses Courses Courses 


PLTW 1 

10 

Al, G 

1CP 

AutoCAD, POE 

PLTW2 

10 

Al, A2 

Biology, 

Physics 

IED, POE 

PLTW 3 

10 

Al, A2, 

Bio, 

IED, POE 



G 

Chem 


PLTW 4 

10 

G, A-AP 


IED, POE 

PLTW 5 

12 

G, A2, T, 


IED, POE, Bio- 



PC, C 


Tech 

PLTW 6 

11 

G 1, G2, 

Bio, 

IED, Design 



A3-A4, 

Chem, 




T, PC 

Phys 


EPICS 1 

10 

Al, G 

Bio, 1CP 

EPICS, IED 

EPICS 2 

12 

Al 

Bio, 

EPICS, 




Zoology, 

Manufacturing, 




1CP 

Communications, 

Construction, 

Robotics, 

Machining 

EPICS 3 

12 

A2, G, 

Bio, 

IED, EPICS 



PC, S 

Chem 


EPICS 4 

12 

Al, A2, 

Bio, 

IED, POE, 



G, PC, C 

Chem, 
Phy, AP 
Phy 

EPICS 

EPICS 5 

12 

Al, A2, 

Bio, 

Manufacturing, 



G, PC, C 

Chem, 

Phy, 

Chem2 

Transportation 

EPICS 6 

12 

Al, A2, 

Bio, 

EPICS 



G, PC,C 

Chem, 

Phy, 

Chem2 
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Math Courses Key: A1-Algebra; A2- Algebra II; G- 
Geometry; S- Statistics; T- Trigonometry; PC- Pre-Calculus; 
C- calculus. 

Science Course Key: Ana- anatomy; Bio- Biology; Chern- 
Chemistry; ICP- Integrated chemistry and Physics; Phy- 
Physics. 

PLTW Course Key- IED- Introduction to Engineering 
Design; POE- Principles of Engineering 

Project Lead The Way Transfer Problem Results 

There were several similarities in the Project Lead The 
Way participant groups when comparing their employed 
cognitive processes during the protocol session. Out of the 
seventeen cognitive processes that were first identified by 
Halfin (1973), the participants only used eight of the processes, 
see Table 3. Due to testing session constraints such as time 
limits, location, and available resources, it was expected that 
cognitive processes such as models/prototypes, measures, and 
testing as well as other cognitive processes would not be used 
when developing a design solution. On average the PLTW 
students used the analysis (AN) cognitive process 37.6 % of 
the time with a range of low 18% to a high of 61%. The design 
(DE) cognitive process was used 36.5% of the time with a low 
21% and a high of 53% of the time. These were the two most 
employed cognitive processes by the PLTW participants. 
Besides the quantitative data that was collected, there were 
several similarities in the ways that the students went about 
solving the problem. 

Of the six PLTW students that were given this ill- 
defined problem, there was distinctive common pattern when 
comparing the two female participant results, see Figure 1.1 & 
1.2 and Table 4 & 5, participants #2 & #5 were females. 
Although the research design did not call for comparison of 
participants’ design thinking by gender, the frequency of the 
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cognitive processes of the female students were similar, where 
as the male students varied. Although these results are 
interesting, the researchers acknowledge that a greater sample 
size (n) would be necessary to generalize the finding. 


Table 3. Halfin Code (1973) 


■ AN 

Analyzing 

■ CO 

Computing 

■ DE 

Designing 

■ OF 

Defining Problems 

■ ID 

Interpreting Data 

■ MA 

Managing 

■ MO 

Modeling 

■ PR 

Predicting Results 

■ QH 

Questioning/ hypothesis 



Fig. 1.1: Project Lead Way Participating School #1 Time Breakdown 
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EPICS High Transfer Problem Results 

With the exception of EPICS High participant #3, all 
other EPICS High students spent 1/3 or more of their time 
designing solutions with a high of 46 % time on (DE) 
Designing for EPICS participant # 6 to a low of 20% EPICS 
participant #3. On average, each EPICS High participant used 
seven of the nine Halfin coded cognitive strategies that were 
employed. In general the EPICS High students not only 
employed multiple cognitive strategies but also spent an ample 
percentage of time employing a variety of strategies. EPICS 
#1 and EPICS # 6 are the two female participants. See figures 
1.3 & 1.4 and Tables 6 & 7. 
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EPICS High School 2 


■ AN 

■ CO 

■ DE 
I OF 

■ 10 
IMA 

■ MO 

■ PR 
QH 





EPICS 4 


EPICS 5 


Fig. 1.4: EPICS High Participating School #2 Time Breakdown 


PLTWresults compared with EPICS High results 

The results of the EPICS High transfer problem 
observational protocol sessions reveal that in general the EPIC 
High students used a greater variety of cognitive capabilities 
when designing a solution to the transfer problem compared 
with PLTW transfer problem results. Clearly, DE (Designing) 
and AN (Analysis) were the most often employed cognitive 
strategies for both groups, EPICS High participants employed a 
variety of other cognitive strategies for an ample amount of 
time, possibly indicating that the EPICS High students engaged 
in a more well-rounded approach to design by employing 
multiple cognitive strategies. Both groups with the exception 
of one participant in each program (PLTW #3 and EPICS #3) 
employed 1/3 or more percentage of their time on the transfer 
problem in DE (Designing). This is a very promising result 
because it indicates that the students were successfully able to 
move from the problem space to the solution space. However, 
4 out of the 6 PLTW participants focused 40% or more of their 
time on (AN) Analyzing. One EPICS High student spent 34% 
of time on (AN) Analyzing, all other EPICS High participants 
dedicated less than 30% of their time using Analyzing as a 
cognitive strategy. It is important to note that (AN) Analysis 
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Halfin code was recorded by the researchers when participants 
were observed breaking down the problem and identifying 
constraints and criteria. (AN) Analysis along with (DF) 
Defining Problems are cognitive strategies that represent the 
‘problem space’ of design thinking. Lawson (1979), identified 
problem solving strategies as either ‘problem focused’ or 
‘solution focused’, and claimed the ‘solution focused’ 
strategies were more representative of a design-based problem 
solving strategy. 

Only PLTW participants #2 and #3 appear to be stuck 
in the problem space, a common pattern for novice designers 
that limits their ability to generate solutions (Cross & Dorst, 
1999; Dorst & Cross, 2001). The PLTW participants that place 
emphasis on (AN) Analysis may be perfonning this way 
because the PLTW students have had experience in identifying 
constraints and criteria in their PLTW course work and 
classroom activities. For example, the PLTW Principles of 
Engineering project, the marble sorter activity (see description 
above in classroom observation section of the findings) require 
the student teams to troubleshoot problems that occur in their 
design solutions by identifying what constraints and criteria are 
limiting the success of the device. This result of this study 
reveals that these PLTW students are possibly transferring their 
knowledge and experience from the Principles of Engineering 
class to the transfer problem sessions. Although these results 
are promising to illustrate a curriculum program having impact 
on students ability to transfer their learning to an open-ended 
transfer problem, too much emphasis of (AN) Analysis can 
limit students’ abilities to move from problem space into 
solution space (Cross & Dorst, 1999; Dorst & Cross, 2001). 

Limited Mathematical Thinking 

Both EPICS High and PLTW student participating in 
the transfer problem observational protocol sessions had 
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limited employment of (CO) Computing. The Halfin code 
computing is defined as “The process of selecting and applying 
mathematical symbols, operations, and processes to describe, 
estimate, calculate, quantity, relate, and/or evaluate in the real 
or abstract numerical sense”. The researchers did not expect 
that participants were going to be generating multiple 
mathematical models to develop solutions to this transfer 
problem in a 15 -30 minute test session; however, the transfer 
problem was created purposely with numbers embedded within 
the problem to see if students would attempt to quantify, 
estimate, calculate, or describe design solutions using the 
numerical infonnation provided within the open-ended 
problem. None of the six participants spent more than 3% of 
their design thinking time using the (CO) Computing strategy. 
PLTW #4 spent the greatest amount of time computing, 
dedicating 3% of time on this strategy, five other participants 
(PLTW #1, 3, 6 and EPICS #1,6) never employed computing 
once during the testing session. Like the results of a similar 
study, Kelley (2008), found that often the participants with the 
least amount of math instruction employed the most 
mathematical thinking. PLTW # 4 had only two math courses 
(Geometry and AP Algebra) but applied the most mathematical 
thinking for 3% of time compared with PLTW 5, PLTW 6, 
EPICS 4, EPIC 5, and EPIC 6 that had five or more math 
courses but employed (CO) Computing for 1 or less % of time. 
In fact, PLTW 6 had taken six math courses and EPICS 6 had 
taken five courses and neither participant employed any 
mathematical thinking during the protocol session. Some 
leaders in engineering design based instruction at the secondary 
education level suggest that mathematical modeling and 
mathematical analysis are key missing pieces in the 
technological design process (Hailey, et ah, 2005; Wicklein, 
2006). Although all student participants have taken advanced 
coursework in mathematics ranging from Algebra 1 to AP 
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Calculas, the participants’ employment of mathematical 
strategies was minimal. This is an extremely important finding 
that could limit supporters of PLTW and EPICS High to 
promote these curriculum projects as effective strategies for 
improving STEM education. 

Design Example Fixation 

As stated above, the students were given an ill-defined 
problem presented on a one page handout. On the problem 
statement handout, there was a stock photo of a school under 
construction and its surrounding areas; see appendix A. The 
photo was used to provide a general context to the problem 
statement. All of the participants at some time in the testing 
session referenced the photo and sometimes analyzed the photo 
even though the photo was never referenced in the problem 
statement. Each participant at some point in the testing session 
used the photo like a template for the location of their 
playground. This discovery was unexpected; however the 
phenomenon has been discovered in other design studies 
(Smith, Ward, & Schumacher, 1993). Smith, et al. (1993) have 
discovered that providing a designer, both novice and expert 
with a design sample can cause the designer to become fixated 
on the design example, thus, limit and influence the designer. 
Kelley (2008) also found that participants in a similar protocol 
study were often fixated on the picture that was included in 
problem statement. Some participants in that study even asked 
questions about the picture and carefully studied the picture’s 
URL citation. 

In this study, students were potentially limited in their 
creative thinking to the context of the photo of a school under 
construction. This can be an extremely important discovery for 
K-12 engineering educators because it illustrates the potential 
negative impact of providing an existing design example to 
students before they generate their own design ideas or even to 



26 JOURNAL OF STEM TEACHER EDUCATION 


provide images or illustrations representing the problem. 
Furthennore, this result of the study may indicate that students 
often believe that solutions to assigned problems are contained 
within the student handout. 


Table 4. Project Lead The Way School #1 Transfer Problem 
Results 



PL1 

rw#i 

PLTW #2 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

10 

01:38.4 

13 

05:48.5 

CO 

0 

00:00.0 

1 

00:08.4 

DE 

8 

03:00.7 

8 

03:14.2 

DF 

6 

02:00.0 

1 

00:44.1 

ID 

1 

00:03.8 

0 

00:00.0 

MA 

0 

00:00.0 

1 

00:02.8 

MO 

8 

02:12.4 

1 

00:13.6 

PR 

2 

00:09.5 

3 

00:25.1 

QH 

0 

00:00.0 

2 

00:15.0 



PLTW #3 

PLTW School 1 Mean 

Scores 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

11 

07:40.3 

11.3 

05:02.4 

CO 

0 

00:00.0 

0.3 

00:02.8 

DE 

8 

02:39.7 

8.0 

02:58.2 

DF 

1 

01:02.8 

2.7 

01:15.6 

ID 

2 

00:11.6 

1.0 

00:05.1 

MA 

2 

00:03.4 

1.0 

00:02.1 

MO 

1 

00:31.1 

3.3 

00:59.0 

PR 

4 

00:21.4 

3.0 

00:18.7 

QH 

1 

00:01.5 

1.0 

00:05.5 
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Table 5. Project Lead The Way School #2 Transfer Problem 
Results 



PL1 

rw#4 

PLTW #5 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

15 

02:56.2 

34 

11:17.1 

CO 

3 

00:27.9 

2 

00:09.6 

DE 

17 

06:47.2 

32 

08:06.5 

DF 

2 

01:21.1 

3 

00:59.2 

ID 

5 

00:38.6 

8 

00:51.8 

MA 

2 

00:09.0 

3 

00:21.8 

MO 

8 

02:08.7 

6 

00:50.6 

PR 

2 

00:05.6 

8 

00:38.9 

QH 

0 

00:00.0 

0 

00:00.0 



PLTW #6 

PLTW School 2 Mean 
Scores 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

11 

04:19.3 

20.0 

06:10.9 

CO 

0 

00:00.0 

1.7 

00:12.5 

DE 

21 

09:07.5 

23.3 

08:00.4 

DF 

2 

00:58.6 

2.3 

01:06.3 

ID 

0 

00:00.0 

4.3 

00:30.1 

MA 

1 

00:12.7 

2.0 

00:14.5 

MO 

14 

02:00.3 

9.3 

01:39.9 

PR 

1 

00:03.9 

3.7 

00:16.1 

OH 

2 

00:32.6 

0.7 

00:10.9 
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Table 6. EPICS High School #1 Transfer Problem Results 



EPICS #1 

El 

PICS #2 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

4 

02:36.4 

9 

01:52.1 

CO 

0 

00:00.0 

1 

00:04.5 

DE 

5 

02:26.6 

14 

03:31.8 

DF 

1 

01:01.2 

4 

01:39.1 

ID 

0 

00:00.0 

2 

00:11.4 

MA 

1 

00:04.5 

2 

00:05.9 

MO 

3 

00:44.4 

3 

00:32.8 

PR 

0 

00:00.0 

4 

00:49.9 

OH 

4 

00:49.4 

1 

00:22.3 



EPICS #3 

EPICS School 1 Mean 
Scores 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

14 

04:08.8 

9.0 

02:52.4 

CO 

1 

00:18.9 

0.7 

00:07.8 

DE 

14 

02:53.0 

11.0 

02:57.1 

DF 

7 

01:48.5 

4.0 

01:29.6 

ID 

6 

00:57.9 

2.7 

00:23.1 

MA 

1 

00:08.9 

1.3 

00:06.4 

MO 

12 

03:40.0 

6.0 

01:39.1 

PR 

1 

00:12.7 

1.7 

00:20.9 

QH 

0 

00:00.0 

1.7 

00:23.9 
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Table 7. EPICS High School #2 Transfer Problem Results 



EPICS #4 

EPICS #5 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

6 

00:40.9 

5 

00:45.4 

CO 

1 

00:03.5 

1 

00:04.4 

DE 

10 

02:34.4 

7 

01:57.8 

DF 

3 

01:20.3 

1 

00:49.4 

ID 

0 

00:00.0 

0 

00 . 00.0 

MA 

1 

00:04.7 

1 

00:13.3 

MO 

4 

00:32.8 

3 

00:27.0 

PR 

3 

00:15.1 

3 

00:13.0 

OH 

0 

00:00.0 

2 

00:13.8 



EPICS #6 

EPICS School 2 
Mean Scores 

Halfin 

Code 

Freq. 

Time 

Freq. 

Time 

AN 

5 

01:21.6 

5.3 

00:56.0 

CO 

0 

00:00.0 

0.7 

00:02.6 

DE 

8 

02:40.8 

8.3 

02:24.3 

DF 

2 

01:07.1 

2.0 

01:05.6 

ID 

0 

00:00.0 

0.0 

00:00.0 

MA 

0 

00:00.0 

0.7 

00:06.0 

MO 

6 

01:13.7 

2.7 

00:44.5 

PR 

4 

00:06.0 

3.3 

00:11.4 

OH 

1 

00:36.4 

1.0 

00:16.7 


Inter-rater Reliability Results 

Two researchers were used to code the transfer problem 
observational protocol sessions. The two researchers were 
trained by the PI of this study on how to interpret the Halfin 
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code and how to record the codes using the OPTEMP software. 
The researchers practiced coding protocol sessions using 
sample videotape sessions as a part of their data analysis 
training. Both research assistants were present for all transfer 
problem sessions and each researcher coded independently 
using the OPTEM software. The researchers hope for 
correlation close to 1, indicating that both raters agree and 
affirm the reliability of these results. The lowest Pearson 
correlations were .76 and .75 for PLTW #1 and PLTW #2, 
these were the first two transfer problem sessions that were 
coded by the research assistants, see Table 8. It is logical that 
the researchers were refining their coding abilities during these 
first two sessions. However all other correlations were .93 or 
higher indicating a strong correlation and reliable coding of the 
transfer problem sessions. 

Limitations 

The researchers acknowledge the study has a small n of 
approximately 60 students observed in PLTW and EPICS High 
classes and an n of 12 participants for the observational 
protocol; therefore, the result of the study are not generalizable 
to all secondary engineering design programs. 

Other possible limitations include a potential bias 
regarding PLTW and EPICS High. According to Merriam 
(2001) researchers must acknowledge potential biases within 
the final report of the study. The researchers acknowledge that 
a potential bias could exist regarding both PLTW and EPICS 
High due to the fact that the researchers work within an 
engineering/ technology teacher program that provides 
undergraduate students with certification in PLTW courses and 
the fact that EPICS High was created by Purdue engineering 
education faculty. To help ensure that such biases did not taint 
research findings, the researchers implemented methods such 
as Merriam’s observational elements guide (2001). The 
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observational elements guide provided researchers with 
guiding questions that required reflection on a variety of 
factors in order to maintain an objective perspective as the 
researcher collected observational data. The researchers also 
use triangulation methods to provide multiple perspectives and 
multiple forms of data. Finally, researchers used multiple 
raters on the observational protocol transfer problem session to 
maintain consistent and reliable data collection. These are 
sound research methodologies that can help to ensure an 
objective data collection process. 


Table 8. Inter-rater Reliability- Pearson R results 


PLTW 

participants 

Pearson 

R 

EPICS High 
Participants 

Pearson 

R 

PLTW #1 

0.76 

EPICS High #1 

0.98 

PLTW #2 

0.75 

EPICS High#2 

0.99 

PLTW #3 

0.92 

EPICS High #3 

0.99 

PLTW #4 

0.93 

EPICS High #4 

0.94 

PLTW#5 

0.99 

EPICS High #5 

0.94 

PLTW#6 

0.93 

EPICS High #6 

0.95 


Conclusions / Implications 

The researchers acknowledge that classroom 
observations were limited to approximately 60 students 
observed in PLTW and EPICS High classes and a sample of 12 
participants for the observational protocol; therefore, the result 
of the study are not generalizable to all secondary engineering 
design programs. However, all members of the STEM 
education community should carefully consider the results of 
the findings from this research. The results of this study 
indicate that students from both engineering design curriculum 



32 JOURNAL OF STEM TEACHER EDUCATION 


programs successfully develop problem solving abilities to 
move from ‘problem space’ to ‘solution space’ as they worked 
through an open-ended ill-defined problem. Furthermore, 
classroom observation revealed that student design teams had 
healthy design discussions and worked cooperatively to solve 
technical problems. 

However, students were limited in their ability to use 
mathematical thinking as a design tool to help create design 
solutions. Although most of the students had advanced math 
classes with over 50% having 3 or more classes (four 
participants had a total of five math classes), students spent 3% 
or less time using mathematics in their design thinking. This 
finding confirms prior findings in similar research. Kelley 
(2008), a study of two approaches to design instruction also 
used a protocol session with a transfer problem found that 
mathematical thinking was a limited cognitive strategy 
employed by the protocol session participants. Moreover, 
Kelley and Wicklein in a national descriptive study of 
engineering design curriculum content found a low emphasis 
on mathematics and engineering sciences in current technology 
education curriculum (2009a). Technology education teachers 
also place a low emphasis on mathematics in student project 
assessment; the individual item using mathematics to optimize 
and predict design results yielded the lowest percentage of 
assessment time (Kelley & Wicklein, 2009b). It appears that 
technology education teachers are not placing great emphasis 
on using mathematical models to predict design results. 
Technology education teachers appear to recognize this 
limitation and have identified it as a teacher challenge. Kelley 
and Wicklein (in-press) in the national status study of 
technology education regarding engineering design also found 
technology education teachers identified: integrating the 
appropriate levels of mathematics and science into 
instructional content (mean of 2.49 on a four-point Likert) as a 
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major challenge implementing engineering design content. 
This was the highest mean score for an individual teacher 
challenge item, thus indicating this is major concern for 
technology education teachers. More efforts need to be placed 
on identifying appropriate mathematical models and science 
inquiry examples that teachers can use to integrate STEM 
concepts in engineering design curriculum content. EPICS 
High teacher #2 provided some insight on why students had 
limited use of mathematics in the transfer problem in response 
to the question: 

“Do you believe that your students will employ 
mathematical thinking (when appropriate) as they work 
to solve the transfer problem?” 

EPICS High teacher #2’s response: 

“Yes, I have given similar design problems that 
required students to use math to solve the problem. 
However, if students can see that they don’t need the 
math to begin designing then I think they won’t use 
math....if they can find a way to solve the problem 
without math.. .they will.” 

These research findings should be considered by 
educators and curriculum developers of pre-engineering or 
secondary engineering design curriculum, of which many also 
promote these programs as ideal platforms for STEM 
education. These findings indicate that currently the transfer of 
STEM learning through the engineering design process on ill- 
defined problem solving is limited. Engineering design 
curriculum developers must be more purposeful in creating 
learning experiences that embed mathematical problems and 
science inquiry activities into the engineering design process. 
More efforts need to take place to locate appropriate ill-defined 
problems that can be explored through the engineering design 
process that will authentically engage students in the analysis 
and optimization stages of the engineering design process. 
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Furthermore, greater efforts must take place across the nation 
to provide professional development opportunities to assist 
math, science, and technology education teachers to locate 
appropriate levels of STEM subjects for delivery in pre¬ 
engineering or secondary engineering design curriculum. 

Future Studies 

The researchers suggest the following studies for the future 
that can extend the results of this research: 

This research study has provided great insights into several 
approaches to teaching engineering design content at the 
secondary level. From this study, other practitioners in the 
field of technology education will better understand what is 
taking place in secondary education classrooms regarding the 
teaching of engineering design. However, more infonnation is 
needed to help properly inform the field about this construct. 
Consequently, the following recommendations are suggested 
for further research to inform the field of technology education: 

a) Larger n for observational protocol studies 

Great insight has been obtained from conducting 
a study to extend the results of prior work (Kelley, 
2008). The sample size of 12 students for the transfer 
problem nearly doubled the prior work (Kelley, 2008); 
however a sample size of 12 is too small to generalize 
to the population. More observational protocol studies 
need to be conducted on students engaged in design 
activities that have authentic levels of math and science 
inquiry embedded into the design problems. This type 
of study can help locate an appropriate balance of 
cognitive capabilities that will have the greatest impact 
on STEM abilities and produce effective designers and 
technical problem solvers. 
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b) Expert designer studies 

The field of K-12 engineering education can 
learn from expert engineers and expert designers 
through observational protocol studies of these experts 
working through a transfer problem. A study of this 
type will help educational leaders in K-12 engineering 
education detennine possible appropriate levels of 
various cogitative capabilities employed at an expert 
level. Currently, the researchers can only indicate 
based upon personal opinion that less than 3% of a 
student’s engineering design thinking dedicated to 
mathematical thinking (CO) Computing is not at an 
acceptable level. Expert designer observational studies 
can provide great insight into what percentage of time 
on various cognitive capabilities indicate an individual 
is functioning as an expert designer. 

c) Revisit Halfin’s study 

Harold Halfin (1973) conducted his Delphi study of 
10 expert technical problem solvers back in 1973, 
others have revised the study before Wicklein and 
Rojewski, 1999 and Hill, 1997 but now is an 
appropriate time to revisit this research and locate 
expert technical problem solvers that represent the type 
of designers that employ the engineering design process 
but also exemplify the type of professionals who would 
have credentials to function effectively in today’s 
global society, see Friedmen,2005; Pink, 2005. Results 
from a study using Halfin’s study as a model could help 
leaders in K-12 engineering education identify 
appropriate competencies for secondary engineering 
education. 
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Appendix A. Transfer Problem 



Problem 


A new K-5 elementary school has been constructed in 
the local area and new playground still needs to be 
designed. You obtained the following specifications: 

The school is expected to have around 500 students. 
The area of the playground has not been determined 
and space is limited. Safety of both students and school 
properties needs to be considered. For example, if 
students are playing softball too close to a building, 
they would risk breaking windows. At minimum, one 
entire grade will use the playground space at once. The 
playground needs to be attractive and fun for all 
students. 

Your Task 

Describe how you would design the playground for the 
school in the problem statement. Please describe all 
assumptions, information you need to obtain, and 
justification of use of space as you “think aloud” your 
strategies for developing the solution. 



